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Background:  Critical  Needs  of  Automation, 
Validation,  and  Discovery 
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=  Sponsor  Activity 


1=  JCIDS  Document 


JCIDS  Process  and  Acquisition  Decisions 


OA 


=  Acquisition  decision 


(J-8  CJCSI  3170.01G)(JCIDS,  2009) 


Multiple  Portfolio  Views: 

•  Systems  vs.  Capabilities 

•  Investment  vs.  Capabilities 

•  System  Context 

•  Highly  dependent  programs 
(Joint  Enablers) 

•  Procurement  Optimization 

•  S&T  vs.  future  needs 

•  Sustainment  Efficiency 

•  Market  Value 


•  Data  are  too  voluminous, 
unformatted,  and  unstructured! 

•  Need  to  leverage  automation 

-  Extract  relations  among  PE, 
MDAP,  and  ACATII 

-  Extract  costs 


DoD  Acquisition  Decision  Making 


OISE 
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Why  and  whatto  buy: 
mission  success  at 
lowest  cost,  service 
advocacy 


Acquisition 

Final  Products  for 

Procurements 

Howto  buy:  balancing 
cost,  schedule, 
performance 


DoD  Budget  Processes2 


How  much  and  when  to 
buy:  control,  oversight, 
balance 


LLA  automates  the  possibility  to  develop  awareness  of  the 
"fit"  between  the  budget,  final  products,  and  requirements. 

1e.g.  Universal  JointTasks  Lists.  Urgent  Needs  Statements.  2e.g-  Program  Elements,  3e.g.  Weapon  Books 
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Methods 


DtSE 


•  Lexical  Link  Analysis  (LLA)  Core 

-  LLA  Reports  and  Visualizations 

•  Collaborative  Learning  Agents  (CLA)  for 
pattern  recognition  that  scales  up  to  Big  Data 

•  System  Self-Awareness  (SSA) 


•  Big  data  and  Deep  Learning  (BDDL)  / 

Big  Data  Architecture  and  Analytics  (BDDA) 


Lexical  Link  Analysis  (LLA)  Core 
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Language-Independent  Processing  Steps 

1 .  Extract  word  pairs  (bigram)  in  context, 
connect  them  as  word  networks  (text-as- 
networks) 

2.  Group  word  networks  into  clusters  /themes 
(by  color) 

Optional  -  Language-dependent  Pre¬ 
processing  Steps* 

1 .  Stop-word  Removal:  Leave  out  stop  words 
(e.g.,  a,  of,  the) 

2.  Named  Entity  Extraction:  Leave  in/out 
people,  places  and  organizations 

•  Focus  on  semantics 

•  Use  NE  in  social  networks 

3.  Parts  of  Speech  Tagging  (Manning  & 
Schutze,1999):  Use  only  nouns  and  verbs 

4.  Stemming:  Use  only  word  roots 


*  Source:  Stanford  NLP  (SNLP,  2012),  Apache  openNLP 


Applying  LLA  as  a  Data-Driven  Automation  % 

Technology  and  Methodology  across  the  DoD  fist 
Acquisition  Process 


•  Surface  themes  and  their  relationships  across 
multiple  data  sources 

•  Compare/correlate  data  from  multiple  data 
sources 


•  Sort/rank  important  and  interesting  information 

•  Discover  high  value  areas  for  investment 
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Research  Progress  2015 
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•  Task  1:  Work  with  the  OUSD(AT&L)  to  install 
the  LLA/CLA/SSA  system  as  a  web  service 
in  the  Defense  Acquisition  Visibility 
Environment  (DAVE)  test  bed  via  the  AT&L 
eBusiness  Center  Service. 

-  Approved  and  in  progress. 
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Defense  Acquisition  Visibility  * 

Environment  (DAVE) 


•  DAVE  interfaces  with  authoritative  data  sources  to  support  the  OUSD(AT&L) 
and  his  staff  in  decision-making,  oversight  and  analysis  for  major  programs 

•  Sample  data  sources 


-  Defense  Technical  Information  -  Acquisition  Information  Repository 

Center  (DTIC)  (AIR) 


o  Program  Elements  (PEs): 

rhttp://www.dtic.mil/descriptivesum/1 

Defense  Acquisition  Management 
Information  Retrieval  (DAMIR) 

(http://www.acq.osd.mil/damir/) 

o  Selected  Acquisition  Report  (SAR) 

o  Defense  Acquisition  Executive 
Summary  (DAES) 

o  Acquisition  Program  Baseline  (APB) 


Test  &  Evaluation  Master  Plan  (TEMP) 

Systems  Engineering  Plan  (SEP) 

Life  Cycle  Sustainment  Plan  (LCSP) 

Acquisition  Strategy  Reports  (ASR) 

Acquisition  Decision  Memorandum 
(ADM,  for  Milestone  B  2366b 
Certification) 

Technology  Readiness  Assessment 
(TRA),  and  LCSP 
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Example  LLA  Reports  and 
Visualizations 
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Big  Data  and  Deep  Learning  (BDDL) 
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•  Task  2:  We  are  also  exploring  how  to  use  LLA  jointly 
with  other  business  intelligence  tools  especially  Big 
Data  Architecture  and  Analytics  (BDAA)  tools 
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Data  Storage  New  Tools 

-  Safe  and  fault-tolerant 

-  SQL  vs.  NoSQL 

•  Graph  databases 

•  Unstructured  data 

•  Cloud  infrastructures 

Operational  Systems 

-  real-time 

-  no- or  low-latency  of  response 
Analytics 

-  Map/Reduce  paradigm  related  to  data 
sciences 

»  Map:  parallel  computing 

•  Reduce:  data  fusion 

-  Apache  Mahout,  Spark 


Challenge 

—  Traditional  data 
sciences  algorithms 

•  Tightly  coupled 
Map/Reduce 
computations 

—  Need  to  augment 
traditional  data 
sciences  with  the 
“Map/Reduce" 
paradigm  to  take 
advantage  of  massive 
distributed  parallel 
computing 


Big  Data  and  Deep  Learning 
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Commercial  Trends  for 
Analytics 


Predictive  analytics 

-  Identify  high-valued  targets 
Graph  analytics 

-  Core  driver  for  Big  Data  analytics 
Text  analysis 

-  Data-driven  discovery  of  themes  and  concepts 
Recommendation  engine 

-  Recommend  high-valued  targets  explicitly 
Deep  learning  (especially  useful  for  CID) 

-  Self-taught  learning:  e.g.  discover/identify  interesting  objects  (e.g.  cats  and  dogs)  in  massive 
images  and  videos 

(http://www.nvtimes.com/2012/06/26/technoloev/in-a-biK-network-of-computers-evidence-of- 

machine-learning.html) 

-  Unsupervised  machine  learning 

•  Pattern  recognition 

•  Anomaly  detection 

-  Data  fusion 

•  Abundant  DoD  applications 


Zhao,  Y.,  MacKinnon,  D.J.,  and  Gallup,  S.P.  (August,  2015).  Big 
Data  and  Deep  Learning  for  Understanding  DoD  Data.  In  the 
CrossTalk ,  the  Journal  of  Defense  Software  Engineering,  Special 
Issue  “Data  Mining  and  Metrics.” 
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Collaborative  Learning  Agent  (CLA): 

It  is  Big  Data  and  Deep  Learning! 


Agent  2: 
Database  2 


Agent  3: 
Database  3 


Learning  Agent 


Anomalies 


Patterns 


LLA  is  Deep  Learning 

•  Pattern  Recognition 

•  Anomaly  Detection 

•  Recursive  Data  Fusion 


Real-time  Data 


CLA  Network 


Normal 


A  Single  Collaborative  Learning  Agent 
(CLA)  Patterns  are  graded  from 
medium  to  relevant  correlations. 


Anomaly  Meter 

Measured  as:  Anomaly, 
Medium  Correlation, 
or  Relevant. 


Value:  10 


Unexpected 


Agent  Collaboration 


System  and  Method  for  Knowledge  Pattern  Search  from  Networked  Agents  (US  patent  8,903,756) 

Zhao  et  ah,  Quantum  Intelligence,  Inc.  (www.quantumii.com) 
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How  Big  Data  and  Deep  Learning  can  be  V  ** 
Applied  to  the  DoD  Acquisition  Process 


•  Examine  and  compare  the  important  acquisition 
data  sources  to  gain  business  insights: 


-  Study  Prime  and  subcontractor  relationships 

-  DOT&E  Annual  Reports  for  horizontal  analysis 

-  Compare  DOT&E  and  DAES:  What  did  the  OT  result 
say  and  did  issues  surface? 

•  Did  these  issues  show  up  somewhere  in  DAES  earlier? 


-  Time  series  analysis 

•  Can  LLA  look  at  program  ratings  over  time? 

-  Compare  budget  data  and  contracts  data 
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How  Big  Data  and  Deep  Learning  can  be 
Applied  to  the  DoD  Acquisition  Process 

(cont’d) 


•  In  the  current  acquisition  process,  a  small  delay  or  anomaly  in  a  contract 
negotiation  process  can  have  a  huge  impact  in  its  performance,  therefore 
cost  the  government  a  lot  of  money  downstream. 

•  It  will  be  very  useful  to  apply  BDAA  such  as  LLA  for  pattern  recognition 
and  anomaly  detection  for  these  kind  of  problems  and  make  early 
warnings  and  predictions  to  prevent  the  downstream  risks. 


•  The  Big  Acquisition  Data  might  include  programs'  cost,  SAR,  DAMIR, 
tech  data,  even  outside  economic  environment  data  if  the  access  is 
possible. 

•  The  causes  of  the  deviations  from  the  normal  behaviors  for  the 
programs/contracts  might  be  modeled  using  physics  (e.g.,  fluid  dynamics 
theories). 
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How  Big  Data  and  Deep  Learning  can  be 
Applied  to  the  DoD  Acquisition  Process 

(cont’d) 

•  LLA's  network  perspectives,  social  plays  among  the 
nodes  and  the  System  Self-Awareness  (SSA)  theory 
may  be  used  to  lay  out  the  academic  vigor  for  the 
business  processes,  for  example,  answer  the 
questions 

-  Are  some  nodes  drawn  towards  some  other  nodes  because 
the  other  nodes  are  more  powerful? 

-  Is  the  preferential  attachment  growth  pattern  or  expertise 
growth  pattern  can  be  used  here? 

-  How  are  the  forces  of  the  nodes  modeled  and  mapped  into 


the  social  network  settings  and  actual  business  processes? 
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